This is a general problem in Ruby. Do you know of any reasonable solutions?

The problem here is WhatLanguage in its current form is dependent on words and having all combinations of casing in the word lists is impractical, so we have to normalize them somehow. Is there a better way to do this normalization?

I did some research on that and there is no simple solution (like 1-to-1 mappings covering all scenarios) as long as there are several conditions to be taken into account, and mostly because some of them are locale dependent (see, for example, Character Properties, Case Mappings & Names FAQ).

Thus we get stuck in a circular problem: we need to normalize the string in order to identify the language and ideally the language must be taken into account in this process of normalization.

Although this seems rather disappointing, I really believe results would be greatly improved if we at least performed those simple conversions (i.e., the case folding as specified by Unicode), even disregarding these locale dependent rules.

Of course this casing conversion goes beyond the scope of this library, so I would propose to use an external one. https://github.com/lang/unicode_utils seems to do the trick and appears to be well written, using official specifications from Unicode. We could dynamically define a to_lowercase method which would either delegate this conversion to UnicodeUtils if defined or simply perform this by String#downcase. That way the user could optionally require the aforementioned library and it would not be a dependency. This sounds too ugly?

I concur. The plan for the next version of WhatLanguage mitigates this somewhat as it will include using histograms of Unicode codepoint usage, but this approach may still be useful.

I think your suggestion in the last paragraph makes sense. Do you want to have a quick attempt at it or would you prefer me to look at it?

I'll try something! Thanks

@peterc, any comments on that?

I think it's a nice, gentle, mostly hands-off approach that could work for now, so thanks! I'll merge it in :-)